Learning about Bias?
An Experiment on Paying Attention to News in Russia

Georgiy Syunyaev
g.syunyaev@vanderbilt.edu

Vanderbilt University, WZB Berlin

Anton Shirikov
shirikov@ku.edu

University of Kansas

Motivation

  • Hard propaganda \(\Rightarrow\) Informational autocrats using biased media

  • Most people consume news from outlets that are biased

    • This can mean fake news, selection, or framing
    • More likely in autocracies with state control of the media environment

  • But often they misunderstand or underestimate media bias

  • Question Can we make citizens learn about media bias?

  • …Does this lead to switching to less biased media, and changes in political attitudes?

How can we answer the question

  • Dual-processing theory: Type 1 processing – automaticity and cue, and Type 2 processing – deliberation

  • Media literacy: Activate Type 2 processing via innoculation or prebunking interventions to improve ability to discern biased or false information

    • Can fail or even backfire, e.g. due to motivated reasoning

  • Attentiveness prompts: Ask citizens about important characteristics of information (e.g. whether it is accurate)

    • But propaganda or biased media does not always lie
  • We combine approaches and try to make citizens learn about propaganda media on their own

Our approach

Ask citizens to analyze the topics and sentiment of news headlines by state and (in addition) independent media

…to test whether…

Paying attention to what media outlet covers make citizens:

  1. Learn about media coverage patterns (positive domestic and negative foreign)

  2. Change perceptions media outlet’s bias (state is more biased)

  3. Adjust media consumption (away from state media)

  4. Change political attitudes (more critical of government / policies)

Setting and design

  • Russia: an autocratic regime with state control of media environment

    • The full-scale invasion in Ukraine exacerbated state media control
    • State media is heavily pro-government (main mouthpiece is Rossiya-1)
    • Some more critical media is still available (e.g. RTVI)

    \(\Rightarrow\) Harder to shift media beliefs and consumption

  • Study:

    • 4-wave online panel experiment among adult Russian citizens (2037 in wave 1 \(\Rightarrow\) 1176 in wave 4) enrolled in online panel
    • Simple random assignment to one of three experimental groups with different sets of tasks
    • Content-analysis tasks at the end of survey in waves 1-3 \(\Rightarrow\) more than 2 weeks between tasks and outcome measurement
    • Analyses focus on the endline outcomes

Intervention

  • Watch 4 to 6 short news headline segments per wave from one of three TV channels:

    • Kultura, a non-political TV channel that covers art, theater, architecture, etc.
    • Rossiya-1, the main state-controlled propaganda channel
    • RTVI, a privately owned TV channel with moderately critical editorial policy

    \(\Rightarrow\) In our sample 67% watch Rossiya-1; 3% (!) watch RTVI

  • After each video, six questions asking to count how many times the following was discussed:

    • Situation in Russia in a [negative/positive] tone?
    • Russian federal officials in a [negative/positive] tone?
    • Situation in other countries in a [negative/positive] tone?

Experimental conditions

  • Placebo: Kultura (Date 1) \(\Rightarrow\) Kultura (Date 2) \(\Rightarrow\) Kultura (Date 3) \(\Rightarrow\) Kultura (Date 4) …
  • State: Rossiya-1 (Date 1) \(\Rightarrow\) Kultura (Date 1) \(\Rightarrow\) Rossiya-1 (Date 2) \(\Rightarrow\) Kultura (Date 2) …
  • State + Independent: Order: Rossiya-1 (Date 1) \(\Rightarrow\) RTVI (Date 1) \(\Rightarrow\) Rossiya-1 (Date 2) \(\Rightarrow\) RTVI (Date 2) …

  • Total of 20 videos to code per respondent over the course of a month

Awareness of media coverage patterns

  • More baseline uncertainty about independent media coverage
  • Evidence of learning about coverage patterns
  • Especially for independent media in respective group

Perceptions of media outlets

  • No change in perceptions of state media
  • But improved perceptions of independent media in respective group

Media consumption and preferences

  • Positive shift in self-reported consumption of both Rossiya-1 and RTVI
  • Stronger revealed preference for RTVI in both treatment groups
  • Increased awareness of both pro-government and critical news

Political attitudes

  • Lower support for the government and stronger concerns about Ukraine

  • No clear differences between treatment groups

Mechanism:
Prior government approval

  • Government critics: Increase in perceptions of unbiasedness and preferences for RTVI; Decrease in support for authorities and increase in Ukraine concerns
  • Government supporters: Increase in consumption of Rossiya-1 and RTVI; No changes in political attitudes

Takeaways

  • Changing beliefs about media bias is hard!

  • Citizens do learn about media coverage patterns when prompted to pay attention to it

    • This is true for both critics and supporters of the regime

  • They also do improve perceptions of relatively new media, but do not revise beliefs about media bias of popular propaganda

  • And do not shift away from state media

  • Only more critical citizens consume more independent media and update their political attitudes

Appendix

References

Sample characteristics

Variable Obs Mean SD Min p25 p50 p75 Max
Socio-economic characteristics
Age 1176 44.81 12.06 19 36.00 45.00 54.00 80.00
Female 1175 0.48 0.50 0 0.00 0.00 1.00 1.00
Education 1176 2.55 0.68 0 2.00 3.00 3.00 3.00
Income level (categorical) 1176 2.82 0.88 0 2.00 3.00 3.00 5.00
Employed permanently 1176 0.74 0.44 0 0.00 1.00 1.00 1.00
Use of advanced Internet technologies
VPN 1050 0.50 0.50 0 0.00 1.00 1.00 1.00
Total (out of 8) 1176 3.22 2.13 0 2.00 3.00 5.00 8.00
Overall news consumption
Frequency (categorical) 1176 2.74 0.63 0 3.00 3.00 3.00 3.00
From TV 1142 0.80 0.40 0 1.00 1.00 1.00 1.00
From online media 1142 0.73 0.44 0 0.00 1.00 1.00 1.00
Recently consumed news
From Rossiya-1 1175 0.68 0.47 0 0.00 1.00 1.00 1.00
From RTVI 1175 0.04 0.19 0 0.00 0.00 0.00 1.00
Intensity of state media use 1173 0.24 0.18 0 0.12 0.19 0.31 0.88
Intensity of indep. media use 1173 0.07 0.15 0 0.00 0.00 0.11 1.00
Attitudes
Media in Russia unbiased 1176 0.52 0.28 0 0.33 0.67 0.67 1.00
Strength of preference for unbiased news 1176 0.36 0.23 0 0.25 0.25 0.50 0.75
President approval 1173 0.60 0.49 0 0.00 1.00 1.00 1.00
Government focuses too much on
Law and order 1176 0.09 0.29 0 0.00 0.00 0.00 1.00
Foreign affairs 1176 0.12 0.33 0 0.00 0.00 0.00 1.00

Concerns over the design?


  • No pure control? No systematic evidence of Placebo (Kultura) group changes over time

  • Political news push respondents away? No differences in patterns of attrition across treatment groups

  • Less attention to political news? High and similar rates of correct answers about the topic and channel across groups

  • Experimenter demand? No differences across treatment groups in respondents’ assessments of study goals

Media perceptions breakdown

Authorities approval breakdown

Policy satisfaction breakdown

Shirikov, Syunyaev · Learning about Bias?